Papers with communication overhead
FedLFC: Towards Efficient Federated Multilingual Modeling with LoRA-based Language Family Clustering (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing frameworks for multilingual modeling face communication costs and parameter interference conflicts. |
| Approach: | They propose a communication-efficient federated learning framework with low-rank adaptation and language family clustering for Multilingual Modeling (MM) they maintain the weights of the base model, updating the lightweight Low-rank adapt parameters to minimize communication costs. |
| Outcome: | The proposed model outperforms the baseline models in performance and reduces communication overhead. |
Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Design (2025.naacl-long)
Copied to clipboard
| Challenge: | Using Mixture-of-Experts, researchers have found that efficient MoE is difficult to achieve due to two key reasons: imbalanced expert activation and massive communication overhead. |
| Approach: | They propose a collaboration-constrained routing strategy that encourages more specialized expert groups and leverages expert specialization. |
| Outcome: | The proposed approach achieves an average performance improvement of 0.51% and 0.33% on LLaMA-MoE and Qwen-MaE respectively. |
FedSpaLLM: Federated Pruning of Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing pruning methods assume public access to calibration data, which is impractical for privacy-sensitive applications. |
| Approach: | They propose a federated learning framework for pruning LLMs that prunes models locally based on private data while accounting for system heterogeneity and communication efficiency. |
| Outcome: | The proposed framework reduces communication overhead and personalizes pruning process based on client resources in federated settings. |
FedID: Federated Interactive Distillation for Large-Scale Pretraining Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | federated learning (FL) is widely studied in user-related natural language processing (NLP) but its performance is faded by confirmation bias. |
| Approach: | They propose a decentralized learning paradigm that uses labeled data to rectify local models . they propose federated interactive distillation (FedID) to alleviate communication overhead . |
| Outcome: | The proposed framework achieves the best results in homogeneous and heterogeneously federated scenarios. |
AnyMAC: Cascading Flexible Multi-Agent Collaboration via Next-Agent Prediction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for multi-agent collaboration rely on static or graph-based topologies lacking flexibility and adaptability. |
| Approach: | They propose a new framework that rethinks multi-agent coordination through a sequential structure rather than a graph structure. |
| Outcome: | The proposed method achieves superior performance while significantly reducing communication overhead. |
Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are demanding more memory and computational resources . however, these devices typically feature weaker GPUs and stronger CPUs . |
| Approach: | They propose a lossless inference acceleration method that leverages the characteristics of heterogeneous devices and the advantages of speculative decoding. |
| Outcome: | The proposed method achieves speedups ranging from 1.79 to 10.1 across different devices . it uses a draft model on the GPU to perform preliminary predictions, while a target model on CPU validates these outputs . |
EcoLoRA: Communication-Efficient Federated Fine-Tuning of Large Language Models (2025.emnlp-main)
Copied to clipboard
Han Liu, Ruoyao Wen, Srijith Nair, Jia Liu, Wenjing Lou, Chongjie Zhang, William Yeoh, Yevgeniy Vorobeychik, Ning Zhang
| Challenge: | Recurrent exchange of model updates in FL can result in prohibitively high communication costs, hindering the distributed learning process. |
| Approach: | They propose a federated fine-tuning framework that uses a round-robin segment sharing scheme to reduce network bandwidth and adaptive sparsification methods tailored to LoRA’s training dynamics. |
| Outcome: | The proposed framework reduces communication overhead without compromising performance on question-answering and value-alignment tasks. |
GMFL: Efficient Global Masking for Federated LLM Fine-tuning (2026.acl-long)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) has emerged as a prominent solution to mitigate the communication and computation costs in federated fine-tuning of Large Language Models (LLMs). |
| Approach: | They propose a plug-and-play layer freezing mechanism to integrate with existing federated fine-tuning frameworks. |
| Outcome: | The proposed solution reduces communication overhead and lowers computational costs while preserving the performance of the underlying federated fine-tuning methods. |
Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel in translation and summarization due to the capabilities of transformer architectures. |
| Approach: | They propose to integrate tensorized adapters into model encoder/decoder blocks to improve model adaptability against data heterogeneity. |
| Outcome: | Experiments on large-scale cross-device FL and large-silo FL show that the proposed methods perform on par or even better than existing federated PEFT approaches while reducing communication cost. |
ProToM: Promoting Prosocial Behaviour via Theory of Mind-Informed Feedback (2026.findings-acl)
Copied to clipboard
| Challenge: | ProToM provides targeted, context-sensitive feedback to individual agents, achieving a higher success rate, shorter task completion times, and is consistently preferred by human users. |
| Approach: | They propose a Theory of Mind-informed facilitator that provides targeted, context-sensitive feedback to individual agents. |
| Outcome: | The proposed system provides targeted, context-sensitive feedback to promote prosocial behaviour, even when not directly aligned with one’s own goals. |